DOCUMENT RESUME 



ED 343 950 

AUTHOR 
TITLE 
PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



TM 018 143 

Van der Meij, Hans 

A Critique of Research on Questioning. 
92 

lOp.; Some ideas presented in this paper were 
published in "General Problems of Questioning 
Research" ("Psychological Research," Number 53, 
1991) . 

Information Analyses (070) — Reports - 
Evaluative/Feasibility (142) 

MFOl/PCOl Plus Postage. 

*Classroom Observation Techniques; Educational 
Research; Elementary Secondary Education; Information 
Seeking; *lnquiry; Knowledge Level; *Learning 
Strategies; Literature Reviews; Naturalistic 
Observation; *Questioning Techniques; *Research 
Methodology; Research Problems 
*Empirical Research; *Question Types 



A majority of the empirical studies on questioning 
has focused on the effects of prior knowledge on the number and type 
of questions asked. Research has not been able to find support for 
the frequency hypothesis and the specificity hypothesis in 
questioning. The focus of these studies is briefly criticized, and a 
detailed critical assessment is made of their methods of study. It is 
argued that many studies on questioning have not succeeded in 
creating a realistic context of inquiry. Certain minimum conditions 
are outlined for seating up research on questioning. Also reviewed 
are the problems witn: (1) using the same kinds of questions found in 
natural interactions; (2) creating a naturalistic exchange between 
questioner and respondent; and (3) enabling people to use other 
inquiry strategies besides questionj.iig. It is argued that these 
issues merit attention because of methodological concerns and because 
they are likely to lead to important insights about questioning. A 
38-item list of references is included. (SLD) 



* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



U.S. OCMIITMeNTOr EDUCATION 
Office of EduaHK)nii Re»«irch and improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 



This document has been reproduced as 
received from the person or orQiini2aiion 
Originating it 

n Minoi Changes have been made to improve 
reproduCti'>n quality 



e Points of view or opinions stated in this docu 
ment do not necessarily represent official 
OERl position or policy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

Hams vAk} i>£it Heia^ 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



A critique of research on questioning 



Hans van der Meij 
University of Tv/«nte, Department of Education 
P.O. Box 217, 7500 AE Enschede 
Netherlands 

Running Head: Questioning Research 



Some of the ideas presented in this paper were published in the following article; 
General problems of questioning research, Psychological Research, 1991, 53, 315-318. 



^ BEST COPY ilVAIL/lBLE 



* 



Questioning Research 1 

Abstract 

A majority of the empirical studies on questioning has focused on the effects of prior knowledge 
on the number and type of questions asked. The research has been unable to find support for the 
frequency and the specificity hypothesis in quesUoning. This paper briefly criticizes the focus of 
these studies and gives a detailed critical assessment of their methods of study. It argues that many 
studies on questioning have not succeeded in crcaUng a realistic context of inquiry. Tliis paper 
ouUines certain minimum conditions for setting up such research on quesUoning. A discussion is 
given of the problems with using the same kinds of questions as found in natural interactions, with 
creaUng a naturalisUc exchange between qucsUoncr and respondent, and with enabling people to 
use other inquiry strategies besides questioning. It is argued that these issues r. rit attention not 
only because of methodological concerns but also because they are likely to h id to important 
insights about questioning. 
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It takes skill and moUvation to ask a question. First, a person who generates a question must 
cxpenence a perplexity and feel compelled to find things out. Second, expressing the quesUon 
nghtly IS a difficult communicative task and must be handled carefully in order to avoid having a 
painful social encounter or facing pubUc embarrassment Third, the person must reflect about the 
success of the quesUon asking strategy and the improvements that might be made in future. In 
short, what is sometimes considered a common part of a person's behavioral repertoire is really a 
most remarkable achievement. Asking a question is an expression of will, of effort, of courage of 
committment, of skill, of knowledge and of lack of knowledge (compare Dillon, 1990) 

Researchers have long been intrigued by what happens when someoni asks "a quesUon 
Generally they have found the matter a difficult one to invesUgate. Empirical studies have 
concentrated on the effects of prior knowledge on quesUoning (i.e. number and kinds of questions 
asked). More particularly, the research has tried to verify two hypotheses: a frequency h^Twthesis 
and a specificity hypothesis. Unfortunately, litUe support has been found. This paper suggests that 
this may be due to the fact that researchers have not paid enough attention to important problems 
in creaung a realistic context of inquiry. A close look at these problems reveals several 
deficiencies all of which <are vital components for developing a theory of questioning (compare 
Dillon. 1986, 1990; Hammer, 1981, 1986; Usunov, 1987). The attention given to these problems 
therefore extends the pure methodological concerns. This paper first presents the two dominant 
research hypotheses and meUiods of study. Then, the various problems in creating a realistic 
context of inquiry will be discu.ssed. 



The frequency and specificity hypothesis 

How is it that people arc capable of asking questions about tilings tiiat tiiey do not known? This 
complex question has been at the basis of many studies on questioning. Unfortunately, researchers 
have often uansformcd tiiis quesUon into tiie much simpler notion of "How much prior knowledge 

no^^K ^ ^"'^'^""^ ^° «PP<^?" Po"- e'^a.Tiple, Miyake & Norman 

(1979) have suggested tiiat pr^ople ask tiie most questions when they face a task of moderate 
difficulty, given average prior knowledge. The task then fits tiieir level of knowledge; the 
unknown is a knowable unknown. Too easy or too difficult tasks wiU lead to fewer questions In 
he first condiuon Uiere is no reason for asking questions and in tiie second there is no schema or 
framework tor generating them. ^ncma or 

Utile support has been found for tiiis frequency hypothesis. With the exception of tiicir own 
s^udy and Pjn of AUwinn-^ (,988). all studies have shown Uiat people ask Ire questilTor 

Snl r f ^'^'"'L' ^ Kai^r & MUller.B(^quet, 1981 

W h o«/1' ^""^'''^ ^987, 1989; Nelson-LeGall & Glor^ 

vScheib, 1985; Smitii. Tykodi & Mynatt, 1988; Van der Meij, 1990a, 1990b 1990c) 

As for predicting tiie kinds of qucsiions asked, tiie prevalent view is that there will be fewer 
specific quesuons for more difficult tasks. According to tiiis specificity hypothesis, a novice who 
tries to master a new computer .system is likely to ask questions Uiat are very broad (e.g. "What do 
I do? ), whereas experienced users ask more focused questions (e.g. "How do I merge tiiis file''") 

.1 ,Q«r ?o^L ^fnSS '"PP°" ''yPO'^esis (Allwood & Eliasson, 1988; Flammer et 

al., 1981, 1984; Fuhrer, 1986; Miyake & Norman, 1979; Van def Meij, 1990a 1990b \9Q0c\ 
Only smith, Tykjxii & Mynatt (1988) found some support; however. JoflTj^^^T^, 
firs stage of a learning process. Novices who gained just a bit of competence quickly began 
asking the same kinds of questions as more experienced subjects. 

Creating a realistic context 

In spite of tiie fact tiiat tiie frequency and tiie specificity hypotiiesis have not had much 
experimental support tiiey have hardly been challenged. Miyake & Noiman's (1979) studv for 
example, is still the most widely cited research on Uie question asking frequency. This is probably 
so because people have found tiic idea intuitively appealing tiiat knowing too litUe or knowing too 
much IS detrimental to a.sking a question. Why then has tiic research consistently pointed out 
otherwise.' And why doesn't higher prior knowledge lead to more specific questions? In short 
why has tiie research done so pooriy in predicting question asking in the experimental seuing^ 
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hvariably, researchers have brought forward methodological arguments. For example, they have 
arpied that prior knowledge was not measured adequately, although measures of general ability, of 
verbal intelligence, and of general and specific domain knowledge have been used. Also 
occasionaUy it is reported that the experimental subjects varied too litUe in prior knowledge, o^ 
Uiat the study has not yielded enough questions for detaUed analyses (see Rammer et al 1984- 
Fuhrer, 1986; Miyake & Norman, 1979; Van der Meij, 1990a, 1990b). 

From the standpoint of cognitive psychology the focus on frequency and specificity is a rather 
poor one because the two variables arc rather unspecific. There is much more to look at For 
example, it would seem important that research on quesUoning addresses issues such as the quest 
for declarauve (episodic or conceptual) knowledge versus procedural knowledge, that it studies 
how problems are tiansformed into questions and hypotheses, that it studies how questions are 
sequenced and that it investigates under what conditions people ask deep, complex quesUons rather 
than simple, obvious one (see Scardamalia, & Berciter, 1991). 

In addition to shifUng the focus towards deeper understanding of the cognitive processes that 
underly questioning (and changing the dependent variables) more attenUon should also be paid to 
the inquiry condiuons in the experiments. One way or another, reseaichere try to create a situaUon 

r!lZ ^i" P"'''^ ^ ' '"'l''''^- "^^y '""^^ face at least three major 

problems: (1) The quesUons asked in the experiment should resemble the quesUons asked in 

^rflLn T"" K ^' "^"^^^ ""^^""'^ '° ^"'"8 s"ch that the questions that 

are asked reflect those that are found in a natural siuiaUon; (2) The exchange between quesUone 
and respondent should be realistic. However, it is almost impossible to create both a natural kind 

LCT, nl'l"°' T^"' ""'^r f^^"" quesUoners; (3) Question 

!.SJ \ T ° f """^ information among many. But. almost inevitably, experimental 
nth ofTh.1 Y° "^'"^ with a situation Oiat limits their inquiiy behavior to asking q,«sUons. 
Each of these three issuas will be discussed below. The examples come mosUy, but not 
exclusively, from research on question asking and prior knowledge. 

Using the Same Kinds of Questions 

It is criticaUy important to study the same kinds of quesUons in the experiment as found in a 
natural situation Experiments that use .^ question selection procedure (which is a method in which 

LeS ,1 > NeT^rr^^^^^ ' TT^''' '^'"^ - Net" 

p.!' 3. Nelson-LeGall & Glor-Scheib, 1985. 1986; Trope & Bassok, 1983- Trow Bassok 
& Alon 1984; Van der Meij, 1990a, 1990b, 1990c) should tlWrefore use qu^Uons Sd^He 
natural setting. The miportance of this simple rule of hand can be illustrat^ with the i^hTf 
Snyder (Snyder & Swann. 1978; Snyder. 1981). In their research Snyder and coUeaguLS tha 
TaS a98^rT'T? Ti'' their S7s. T^^"'& 

sd^^^hl n.T^o'^H?'' ""^n ^ °f ^hich the subjects 

an mte view With a new list, based on questions asked in real interviews, Uiey found question 
asking to be diagnostic rather than confirmatory (see also Skov & Sherman 1986) There is 
other evidence of the troublesome nature of the selection proceduTJor exLpL when^ 

co™'"''''" ^' ^^ experiment rf^d UtU 

correspondence on matters such as frequency and type of question asked Van der Meij 1990b) In 
contrast, when I correlated question-asking frequency (i.e. selection) in the ex^r^rem with 
qu^tion-asking frequency in the classroom he found a low but significant conelatio^u " .28. £ 

The selection procedure is not a bad method by nature. It seems, for example, a practical way 

isLer^aSon't T""'^ ^'-^T' Mother with 

mml l^ ! raising (^e. generating). In addition, it bypasses the hard problem of the 
mimediate categor^ation of the questions and of choosing Uie right answer during the experim^^^ 
ff^ amrncr, 1986). However, it does have at least one serious drawback and that I that Tis 
d ficult to setUe upon -he right kinds of questions for the experiment. Unfortunately, research^ 
oftcr, cannot use the na; :rally occurring questions because Uiere are frequently not enough of such 
questions, or because ihere is too liiUc correspondence with Uie experimental task What 

rZonrT??'' "^'""T i! ^"'^ S'"'-^'^ experimental questions that resemble real 
questions. This resemblance is based upon two critical assumptions. First, each question is 
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categorized into a particular class of questions. Second, each class of questions is believed to 
correspond closely to a particular question funcUon (see Nelson-LeGall, 1987; Nelson-LeGall & 
Glor-Scheib, 1985, 1986; Nelson-LeGall, & Jones 1990; Nelson-LeGall, Kratzer, Jones & 
DcCooke. 1990; Van der Meij, 1990a, 1990b, 1990c). For example, Nelson-UGaU & Glor-Scheib 
(1985) classified some children's quesUons (e.g. "How do I solve 14 x 3?" and "Can you show me 
how to mulUply fracUons?") as requests for hints, and others (e.g. "What is 14 x 3?" and "Can you 
tell me the answer?") as requests for answers. Each question class was believed to reflect a certain 
information-seeking function. The hints were considered a good measure of quesUon asking for the 
purpose of learning how to .solve problems. The answers supposedly came from the child's 
intention to complete tasks without much personal involvement. 

Both assumptions may be troublesome. First, categorizing questions into a particular class is a 
tricky business and researchers should probably go beyond getting good interrator agreement 
scores to assure the vaUdity of the coding. Second, it may not be such a good idea to believe that 
there is a one-to-one correspondence between the form and funcUon of a quesUon. Peihaps 
assuming complex relationships is the better opUon. In any case, additional measures to the 
questions being asked should be used to substantiate these presupposed relationships. 

For example, many researchers have tended to concentrate on the information-seeking function 
of questions. In dense social settings such as classrooms tiiis may be the wrong focus. In these 
settings questioning might primarily serve the social purpose of helping pupils to develop 
relationships as tiie following example illusttates (Gumperz, 1981, pp. 18-19): 

"The black children tended to call the teacher or the aide to ask for kelp and even after an 
explanation they were often heard to commeM 7 can't do this' or 'I don't know how to do 
this'. All such sentences were pronounced with similar intonation contours, characterized by 
high pitch register, sustained tone, and vowel elongation on the last syllable. We (...) asked a 
group of black judges whether they thought the children really didn't know or didn't want to 
cooperate. The judges agreed in saying what the children really mean to say in these cases is 
Help me; I don' like to work alone'." 
Ono may thus wonder whetiier tiie two types of questions distinguished by Nelson-LeGall reaUy 
exhaust the ones tiiat arc generally asked in school. For example, Meece, Blumenfeld & Hoyle 
1988 argue for tiie presence of a third major funcUon namely an ego, or social goal (see NeLson- 
LeGall. 1990). Categorizing tiie questions as serving a particular kind of infomiaUon-seeking or 
social function is also difficult (see Usunov. 1987). Should one use syntactical, semantic or 
pragmatic considerations to categorize a question? And is it fair to assume a close conespondence 
between tiie expression, class and function of a question? Researchers must come up witii an 
answer to Uiese questions regardless if tiicy use a selection procedure or whetiier the subjects an 
tree to generate their own quesUons. 

Shaping the exchange 

Laboratory studies on questioning always sUiiggle wiUi tiie exchange between questioner and 
respondent. In tiicse exchanges tiiere arc prolcms with timing, nonn-settiiig, and wiOi tiie content 
of the response. 

Timing is crucial in answering questions and in general respondents should answer questions 
prompUy and not a very long time afterwards. In experiments on Uie effects of prior knowledge on 
questionmg tiiis is problematic since direct responses may activate donnant prior knowledge 
Moreover, die responses arc likely to lead to a unique pattern of responses and follow-up 
questions for each subject and tiiereby seriously compromise comparability between subjects 
llius, if Uic experiment simulates a naturalistic 'question - response - processing of response' type 
of interaction it becomes virtually impossible to analyze the effects of prior knowledge on 
qucsuoning. To solve tills problem researchers have handled tiie timing problem in tiie following 
ways: tiicy have not answered tiie questions at all, or Oiey have asked tiieir subje<'.ts to pose or 
ro"ou ^i"^" ^" °^ questions before answering each individual one (e.g. Allwood, & Eliasson 
1988; Flammcr, Grob, Lcutiiardt & LUtiii, 1984; Miyakc & Nonnan, 1979). These solutions are 
unsausfactory for a number of reasons. First, in tiie absence of direct responses questioners are 
unlikely to be concerned with communicating tiieir precise informational needs. They need not 
create a common ground'. The effect may be an abundance of questions tiiat are difficult to 
mterpret (compare Miyjikc & Norman, 1979). Second, tiie subjects probably become more 
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selective in what they ask and they arc more likely to keep their questions for themselves. When 
this leads to an increase of 'intemar questioning the validity of the study is seriously affected. 
Third, since the subjects cannot base their questions upon responses to previous questions they are 
severely limited in following a plan of action; they simply cannot follow a strategy in which 
questions and responses are integrated. Instead, th y mus* revert to a strategy of asking as many 
questions as possible, finding out only later that a numbci J these questions were useless in view 
of the answers received to critical previous questions. 

All experiments follow the Gricean principle of cooperation. That is, the norms are set in 
favor of question asking and social constraints are minimized to give subjects maximal freedom to 
express questions as they arise. In addition to removing obstructing conditions, the instructions 
often explicitly encourage subjects to ask questions. In short, question asking is not merely 
enabled, it is clearly stimulated by the instructions (e.g. AUwood, & EUasson, 1988; Fishbein, 
Eckart, Lauver, Van Leeuwen, & Langmeyer, 1990; Flammer, Kaiser & LUthi, 1981; Hammer, 
Kaiser & MQUer-Bouquet. 1981). How will this effect question asking? I would speculate that 
these procedures increase the chances that some subjects make up questions to please the 
experimenter. Other subjects may ask only a few questions because they feel a little uneasy about 
taking the floor while a 'superior' -the experimenter- is present (see DiUon, 1988, 1990; Mishler, 
1975, 1978) or because they do not believe that the conditions for question asking are biased in 
their favor -there are ambiguous signs since timing is not arranged at the convenience of the 
subjects-. 

Another problematic part of the exchange lies in answering. How should one determine the 
content of the response? In all interactions there are difficulties with answering. For example, 
what should one do with questions tl t arc based on a false presupposiUon as in the famous 
question "Have you stopped beaUng your wife?" And how should one respond to open questions 
like "What do I have to do?" In all sfjdies the researchers must have formulated some niles of 
conduct However, only Flammer and co-workers have described these rules in detail (see 
Flammer, Kaiser, & LUthi, 1981). According to Flammer, respondents lequirc knowledge about the 
followmg factors. First, they need domain or task-specific knowledge. Second, they must know 
how to classify the questions. Third, they must have a set of principles for selecting the right 
response. In his experiments, especially in the later ones, Flammer worked hard to standardize 
these factors. So he began wiU) drawing a very detailed chart of tiie experimental task (i.e. 
prepanng a mousse-au-chocolat) and presented this chart to the respondents who then all had the 
same task-specific knowledge. Next he categorized the domain into a number of classes (e.g. 
goals, actions, criteria for success, ingredients and instruments). For each domain-class he further 
specified Uie kinds of questions into types (e.g. yes/no. global/specific, one or more 'steps') and 
coupled each to a specific answer rule. So, if an experimental subject asked a question such as 
With what thmg do I do this?" all respondents would classify it as an instniment question of a 
global nature. Followmg the prescribed response rule they would answer "With a rod" 
Unfortunately this important infonnation is left out of the published article of the experiment 
(Flammer et al. 1984). Interestingly, Flammer later (1986) argued Uiat he found himself tangled in 
issues concerning question answering raUier than in question asking in which he was interested. 
He has the good company of researchers in artificial intelligence research (e.g Lehnert. 1978) in 
philosophical research (e.g. Harrah, 1984), and in communication risearch (sec Dillon, 1990) who 
all address answcrhood. In contrast, most of tiie empirical studies on questioning do not describe 
the behavior of the respondents, nor tiie response rules upon which \his is based (e.g. Fuhrcr, 
1989; Fishbein et al., 1990; Smith, Tykodi, & Mynatt, 1988). This is a serious omission. Without 
this information the findings arc virtually worthless because it is hard to imagine what interactions 
nave taken place. 

Enabling other Information Seeking Strategies 

A third problem in cxpcrimenial research on questioning is tiiat one would like to have subjects 
use Uieir full range of infonnaUon-secking strategies in dealing wiUi Uie problem at hand and not 
just quesuon posing. Normally there are lots of ways of gaining infonnation besides question 
posing. For example, people can consult study-guides, dictionaries, encyclopedia and oUicr written 
materials, or they can touch or observe certain phenomcn?. Moreover, they can reflect upon a 
nscn issue for a long time without ever expressing a question. Asking a question to a respondent 
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IS thus but one of many ways in which people search for information. In most experiments, 
however, the subjects can only ask questions to find things out besides guessing and inferencing. 
The researchers have had good reasons for mi.iimizing other inquiry behaviors. As the obstacles to 
asking questions are quite strong, researchers someUmes cannot do otherwise but to stimulate them 
by invitation. Also, it is often tacitly assumed that the expressed questions in the experiments 
reflect (a wide lange oO the questions that undcrly other inquiry behaviors. Thus, if someone, soy, 
consults a document it is presumed that the person has a question in mind for which he or si e 
seeks an answer in the document. However, some questions just are too ideosyncratic, or ill- 
fonned that respondents cannot be expected lo give helpful answers. In such a case people are 
likely to use other inquiry behaviors than expressing a quesUon. More generally then, since the 
subjects in most of the experiments can only ask questions, these studies reveal nothing of the 
relationships of question asking with other kinds of inquiry behavior. 

Conclusion 

In general, research on quesUoning and prior knowledge has not been very succcsful in verifying 
cither the frequency hypothesis, or the specificity hypothesis. Besides iitUe success, the cogniUve 
research on quesUoning also has not paid enough attenUon to issues such as how problems are 
transformed mto questions and how people become committed to a quest for knowledge. 

Most of the published studies on quesUoning also do not pay enough attenUon to Uie problems 
associated wiUi creaUng a realisUc context of inquiry. More attenUon is needed on using Uie right 
kinds of quesUons, wiUi following an exchange Uiat aUows for prompt and standardized responses, 
and wiU) presenUng altcmaUve opUons for informaUon-gaUicring strategies. Some of Uie cited 
studies have addressed a number of Ujese problems, but unfortunately, Uieir systemaUc approach 
can often be detected only by looking at a series of studies covering a long period of research 
This IS, for example, the case wiUi Uie studies of Nelson-LeGall which cover approximately six 
years of research. OccasionaUy too, one finds SL;h informaUon in unpublished, internal reports 
that are difficult to obtain (i.e. Flammer's study). 

The problems menUoned in Uiis paper are not of interest solely because of meUiodological 
concerns. The approaches Uiat researchers take will lurely also contribute towards a deeper 
understanding of quesUoning and advance Uie development of a Uieory of quesUoning. ITieir stance 
with regard to issues such as Uie disUncUon between Uie form and funcUon of quesUons and Uie 
disuncuon between informaUon-seeking and social funcUons of quesUons may inform us about Uie 
problcmaucal topic of defining quesUons. SUuidardizaUon efforts wiUi regard to Uie rules of 
conduct for Uie respondent can yield importar. insights into Uie pragmaUc properties of quesUon- 
answer situauons. Also, knowing someone's reasons for posing a quesUoa as opposed to using 
another means of inquiry would seem to be an important building block in a Uieory of questioning 
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